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MEMORY MANAGER FOR MULTI-MEDIA 
APPARATUS AND METHOD THEREFOR 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

The subject matter of the present application is related to 
that in U.S. patent application Scr, No, 08/856,118 filed May 
14, 1997 entitled "PROGRAMMABLE FOUR-TAP TEX- 
TURE FILTER", now U.S. Pat. No. 5,920,495, and Ser. No. 
08/841,360, filed Apr. 23, 1997 entitled "VIRTUAL 
MEMORY MANAGER FOR MULTI-MEDIA ENGINES", 
now U.S. Pat. No. 5,835,104, both of which are incorporated 
herein by reference. 

FIELD OF THE INVENTION 

The present invention is related to the field of multi-media 
integrated -circuit controllers for use with computer systems. 
In particular, the present invention is directed toward a 
virtual memory manager for use with multi-media control- 
lers. 

BACKGROUND OF THE INVENTION 

Prior art computer systems generally employed a graphics 
controller (e.g., VGA, SGVA, or the like) such as illustrated 
in FIG. 1. FIG. 1 is a block diagram illustrating major 
components of a computer system 100 provided with display 
controller 120 (e.g., Video Graphics Adapter (VGA), Super 
VGA (SVGA) or the like). Display controller 120 may 
generate pixel data for display 180 (e.g., CRT, flat panel 
display or the like) at a rate characteristic of the refresh rate 
of display 180 (e.g., 60 Hz, 72 Hz, 75 Hz, or the like) and 
horizontal and vertical resolution of a display image (e.g., 
640x480 pixels, 1024x768 pixels, 800x600 pixels or the 
like). A continuous stream of pixel data may be generated by 
display controller 120 at the characteristic rate of display 
180. 

Display controller 120 may be provided with a display 
memory 130 which may store an entire frame of pixel data 
in text, graphics, or video modes for output to display 180. 
Host CPU 110 may be coupled to display controller 120 
through bus 150 and may update the contents of display 
memory 130 when a display image for display 180 is to be 
altered. Bus 150 may comprise, for example, a PCI bus or 
the like. System memory 160 may be provided coupled to 
Host CPU 110 for storing data. 

Hardware MPEG decoder 140 may be provided to decode 
MPEG video data from an MPEG video data source (e.g., 
CD-ROM or the like) and output decoded video data to 
system memory 160 or directly to display memory 130. 
However, with the advent of increasingly powerful and 
faster microprocessors (e.g., Pentium™ or PowerPC™ pro- 
cessor or the like) it may be possible to implement MPEG 
decoding (or the like) entirely within software operating 
within host CPU 110. 

In recent years, "multi-media" devices for computers 
have become popular. Multi-media, as the name implies, 
encompasses a number of different types of functions, such 
as 2-D and 3-D graphics, animation, Full Motion Video™, 
audio, and the like. Such devices may be used for games or 
for business use (e.g., teleconferencing). 

Unfortunately, implementation of multi-media technol- 
ogy has been hampered by the lack of any coherent standard 
for various discreet multi-media devices (graphics 
controllers, CD-ROM or DVD controllers, sound cards, 
modems, and the like). As a result, a user may become 
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frustrated to discover that different devices conflict with one 
another or will not work within his or her system. 

Some attempts have been made to provide better com- 
patabilty between multimedia devices. For example, 

5 Microsoft® has promoted the use of Plug and Play™ 
technology. Plug and Play™ is a standard by which various 
peripheral devices may be inserted into a computer system 
which will automatically configure the system and resolve 
any device conflicts. 

10 However, even with such systems, other drawbacks are 
present. For example, to provide realistic three dimensional 
(3-D) imaging, considerable processor power may be 
required. Users may be reluctant to discard existing personal 
computer (PCs) in order to take advantage of new 3-D 

1S software. 

FIG. 2 is a block diagram of one apparatus which attempts 
to solve this problem. The apparatus of FIG. 2 is described 
in more detail in Talisman: Commodity Realtime 3D Graph- 
ics for the PC by Jay Torborg and James T. Kajiya 
(Microsoft Corporation, Aug. 21, 1996) incorporated herein 

20 by reference. In FIG. 2, a single multi -media device may be 
provided as a plug-in card for a PC. The multi-media device 
may be provided with a media signal processor (MSP) 210 
coupled to a system bus 200 such as a Peripheral Commu- 
nications Interface (PCI) or the like. 

25 Media I/O device 230, providing an interface for joysticks 
or the like (e.g., USB 1344 or the like) may be provided 
coupled to MSP 210, along with Audio CODEC (audio 
modem) 240. Polygon Object Processor 250 may render 3-D 
images of polygons. Image layer compositor 260 may com- 

30 bine a number of image layers or elements using a compos- 
iting buffer to generate an output image. 

Graphics memory 270 may comprise a RDRAM or other 
high-speed memory. Video output may pass through COMP/ 
DAC 280 (e.g., decompressor, RAMDAC, and the like) for 

35 color conversion and output to a display. 

One disadvantage of the design of FIG. 2 is that each of 
the various chips may have intensive processing require- 
ments. Graphics rendering chips 250 and 260 may have as 
many gates as a Pentium™ and Pentium™ PRO™ 

40 processor, respectively. Thus, the device of FIG. 2 may be 
relatively complex and expensive. Moreover, the many of 
the functions served by the apparatus of FIG. 2 may be able 
to be performed using more advanced host microprocessors, 
such as the Intel® MMX™ processor or its progeny and 

45 successors. 

FIG. 3 is a block diagram of another approach to an 
improved multi-media controller. In the apparatus of FIG. 3, 
an advanced microprocessor communicates with a multime- 
dia controller 310 over an Accelerated Graphics Port (AGP) 

so 300 which may be a PCI -like type communications bus 
provided with specialized and enhanced communications 
features. Multimedia controller 340 incorporates may of the 
functions of the elements of the apparatus of FIG. 2 and in 
addition may rely upon an enhanced host processor to 

55 perform some multi-media functions. Multi-media control- 
ler 310 may be coupled to RDRAM 320 which may be used 
to store image data. 

The apparatus of FIG. 3 has the advantage of being able 
to interface with newer processor designs. However, the 

60 apparatus of FIG. 2 may be more suitable as an upgrade to 
existing computers. In either design, however, the size of the 
semiconductor devices may become unwieldy and cost 
prohibitive due to the complexity of the designs and the 
functions to be performed. Thus, it remains a requirement in 

65 the art to provide a multi-media controller which preforms 
required multi-media functions with a minimal number of 
gates. 
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One proposal for simplifying graphics controller archi- local memory or system memory) and will also convert the 

tecture is to unify or partially unify display memory with data stream to or from a compressed format, 

main memory of a computer. Referring back to FIG. 1, note \ n addition, the second virtual memory controller pro- 

that display memory 130 comprises an additional memory to v ides a unified TLB (translation lookaside buffer) available 

system memory 160. Some have argued that such a system 5 to all media units. The TLB has four types of pointer entries 

is duplicative, and that unifying the two memories 130 and which are controlled by two bits. The first bit controls 

160 would reduce overall memory requirements and allow whether the TLB entry is a direct map or a pointer to another 

for greater component integration. translation table, the second bit controls whether the TLB 

Such a system may reduce the amount of memory entry is stored in a 'compressed format. The overall concept 

required for a computer system, and thus reduce overall cost 30 may allow cdynamic~I6a^3alalicih^ 

of the computer. Intel® Corporation has proposed a version ^emdry^nd^sysle^^e^oly^ 
of Unified Memory Architecture (UMA) in which display 

memory is formed from a part of system memory. While BRIEF DESCRIPTION OF THE DRAWINGS 

such a system may reduce component count, it may also HG t {s a ^ ^ iUustrati major components 

create bandwidth bottlenecks. 35 c . 4 * 4 P, , J ... *\. , 

of a prior art computer system provided with a display 

In particular, a CRT or other type of display requires a controller 

cons.ant stream of data in order to generate a display image. pjo. 2 is a block diagram of a proposed single multi- 

At h>gher reso unons and refresh rates, thts data rate may be ^ deyke M £ ^ p * 

fairly high. If the display needs to access main memory for r . , 

a significant period of time, such accesses may hamper 20 FIG - 3 is a block 6 ™&* m oi another Proposed approach 

overall CPU performance. To overcome the deficiencies of lo an ™P™ed multi-media controller using an Accelerated 

UMA, the assignee of the present application has proposed Graphics Port. 

a solution known as Partially Unified Memory Architecture FIG. 4 is a block diagram of a present multimedia system. 

or PUMA. FIG. 5 is a block diagram of a media controller imple- 

Reference is made Bril et al., co-pending U.S. patent mentation illustrating two media units. 

application Ser. No. 08/624,128 (0455-VDSK) entitled FIG. 6 is a simplified block diagram of a multimedia 

"PARTIALLY UNIFIED MEMORY ARCHITECTURE", virtual memory mapping system of the present invention. 

now abandoned and incorporated herein by reference. A FIG 7 ^ a 51ock diagram 0 f a multimedia apparatus 

memory controller in a Partially Unified Memory Architec- 3Q which may i ncorp0 rate the present invention. 

ture controls allocation of memory requests between one or 0 . „ , . .„ . 

L 1 c t. , FIG, 8 is a flowchart illustrating the steps in memory 

more memory areas to enhance the performance of a display „ . 1t . ,. . . r . . f(U 

„ * . 1, • i allocation by the multimedia virtual mapping system of the 

controller. The memory controller receives memory cycle J .. 

* r . i * u * i • j • i present invention, 

requests from at least one subsystem including a display r 

controller. A cycle distributor allocates requests to Main 35 DETAILED DESCRIPTION OF THE 

Memory Display Memory and Dedicated Display Memory INVENTION 
sequencers depending on operational modes, memory 

configuration, and BIOS settings related to bandwidth opti- In multimedia systems there may be a number of different 

mization. * data types transferred between a number of different types of 

Bril et al. teaches that it is possible for a subsystem such 40 processing units. Thus, unlike a typical prior art graphics 

as a display controller to share physical memory between controller, a multimedia controller may be more data and 

local (e.g., display memory) and main (e.g., system) data type stream oriented. Examples of different data types 

memory. If UMA or PUMA is to be implemented in a include textures, video, surfaces, command lists, patterns 

multi-media application, memory for a number of different ( font files and the llke )> WaveTable sample spaces, MPEG 

devices (e.g., VGA controller, sound card, MPEG encoder, 45 blocks ( H - 263 or H - 261 lv P e encode and decode blocks), 

and the like) may need to be allocated and controlled on a C0M buffer data > audl ° buffer data > and the uke - 

dynamic basis. A command list or execute buffer is a list of instructions 

Virtual memory mapping for computer systems is known t0 be processed. For example, a particular multimedia unit 

in the art. In so-called "virtual mode" of operation, a CPU engine, whether fixed or programmable, may have a driver 

may swap data between system memory and a hard disk 50 with a list of instructions or commands may be a triangle list 

space. Regardless of whether data is located in system for 3D, a list of BITBLT instructions, a list of audio 

memory or on the hard disk, such memory may be virtually functions to deal with certain streams, or the like, depending 

addressed such that addressing is not dependent on the u P on the Dature of the unit the data processed, 

physical location of the memory. However, such virtual FIG. 4 is a block diagram of a multimedia system. Present 

memory mapping systems have generally been limited to a 55 multimedia systems may be characterized by a software/ 

single device (e.g., CPU or peripheral), and thus each device hardware implementation, which is favored by the 

may require its own virtual memory mapping manager. Microsoft™ operating system and the recent AGP Bus from 

Intel™. In this environment, software based in CPU 430 as 

SUMMARY OF THE INVENTION well ^ software based software in external media controller 

The present invention comprises a virtual memory man- 60 450 (fixed or programmable) both operate on related data, 

ager for a multi-media engine. A second virtual memory Such data may be stored in local media memory 410 or in 

controller is provided unique to the media system. The system memory 420 as space and bandwidth requirements 

second virtual memory controller allows individual media allow. 

units to operate in their own virtual space in much the same Media controller 450 generally does not contain a single 

way as a software program operating in virtual mode. The 65 function or unit. For example, graphics controllers have 

second virtual memory controller performs address transla- advanced lo contain multiple independent units for 2D 

tion or mapping to the correct physical memory location (in graphics, 3D graphics, video capture, video playback, and 
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CRT refresh. Each of these units and its respective software The use of the term "multi- media" which also describes the 

counterpart (i.e., driver) may operate on its own set of data. preferred embodiment and best mode, should not be taken as 

CPU 430 operates on this data through a virtual-to- limiting the invention to only a multi-media environment, 

physical mapping provided by the O/S. However, individual The multi-media virtual memory controller 630 of the 

media units inside media controller 450 may still be man- 5 present invention allows individual media units 610, 620 to 

aged through physical memory pointers to specific system or operate in their own virtual space, much the same way as a 

local memory space. Such a system requires explicitly software program running on an O/S would do today, 

determining such mapping locations, locking them down Multi-media virtual memory controller 630 receives address 

(i.e., fixing them), and programming the individual units. data addr from units 610, 620 and performs an address 

When the O/S decides to move such data, due to bandwidth 10 translation or mapping to the correct physical memory 

or space concerns, the unit must be completely repro- location in local or system memory through local memory 

grammed with new physical memory pointers. control 640 or system memory control 650, respectively. 

In addition, many media units may be rigidly fixed and Multi-media virtual memory controller 630 may also 

may only operate out of data in local memory or system convert the data stream to and from a compressed format if 

memory. Such fixed media units do not allow performance 15 it has been identified as such. Thus, uncompressed data may 

and system resource optimization. Moreover, such data may be stored in a compressed format by block 690 and then 

be stored in a compressed format for both bandwidth and decompressed in block 690 when read by a unit 610, 620. 

memory size issues. In such situations, the media unit may Unit 610, 620 may not be aware of such compression/ 

then be required to be aware of the data format type and decompression activity occurring and thus compression/ 

perform a data format conversion before processing the data. 20 decompression software or hardware is not needed for each 

FIG. 5 is a block diagram of a media controller imple- unit 610, 620. In the prior art, each unit may be provided 

mentation illustrating two media units. For purposes of with its own unique compression/decompression hardware 

illustration, the apparatus of FIG. 5 is shown with two or software, greatly expanding the complexity of a multi- 

" general" media units 510, 520. Each of units 510, 520 may media device, 

have its own mapping and conversion logic. However, in the 25 In addition, block 690 may also monitor caching of data 

example of FIG. 5, media unit B 520 is fixed in its source of in virtual memory. Again, each unit 610,620 may not be 

data. aware such caching takes place, and the use of a centralized 

Media unit B 520 may have fixed address pointers b_s_ caching block 690 eliminates the need for redundant cir- 

addr to a portion of system memory 550. O/S software may 3Q cuitry in each multi- media unit 610,620. 

reprogram media unit B 520 to relocate address pointers Multi-media virtual memory controller 630 is provided 

b_s_addr. However, such programming may not redirect with a centralized and unitized translation lookaside buffer 

data to other memories (e.g., local memory). (TLB) which keeps track of the relationship between physi- 

Media unit A 510 may contain stand-alone virtual cal and virtual memory pointers for all units 610,620. By 

memory mapping unit 530. Virtual memory mapping unit 35 providing a centralized TLB, all units 610, 620 may operate 

530 may create a virtual memory map for media unit A 510 from one TLB, and thus avoid memory conflicts and the like, 

such that media unit A 510 may address data using address If one unit 610,620 (or a host CPU) relocates a block of data, 

pointer addr which in turn is re-mapped to system memory or if multi-media virtual memory controller 630 relocates a 

control 550 or local memory control 540 using address block of data for band width or space considerations, all units 

pointers a_s_addr and a_L_addr, respectively. Media unit 4Q 610, 620 (as well as a host CPU) will still be able to access 

A 510 may thus address data without regard for the location that data without conflict. 

of that data in system or local memory. TLB 635 may have four types of pointer entries which 

However, note that virtual memory mapping unit 530 may be controlled with two bits in the preferred embodi- 

provides memory mapping for only one media unit. ment. However, the number of bits may be expanded within 

FIG. 6 is a simplified block diagram of a virtual mapping 45 the spirit and scope of the present invention. The first bit 

multimedia system of the present invention. In the present may control whether the TLB entry is a direct map or a 

invention, memory space is virtualized away from hardware pointer to another translation table. This first bit allows 

units of a media controller. In this way, each hardware unit complex scatter-gather tables to be used, or to point to a 

610, 620 need not be aware whether memory is access portion of a page table of a host CPU. A scatter-gather table 

through local memory control 640, system memory control 50 allows a block of memory to be divided up (i.e., scattered) 

650, or be compressed, uncompressed, or cached by block within a physical memory if a contiguous block of memory 

690. Load balancing and latency requirements become an ' is not available. 

operating system or device management issue. Simply The second bit may control whether the TLB entry is 

changing or updating pointers in a virtual memory transla- stored in a compressed format. The use of a second bit to 

tion table 635 is all that is required. 55 indicate compression is one of the unique features of the 

The apparatus of FIG. 6 also allows units 610, 620 which present invention and further distinguishes the present 

have traditionally have not had access to other memory invention from prior art virtual memory managers, 

spaces to utilize them. With the band width provided by AGP, In addition, block 690 may also provide a cohesive [ 

this approach provides some unique opportunities and clear technique for caching memory traffic and allowing dynamic! 

advantages over the traditional approach. eo load balancing between local media memory and system * 

The present invention uses a multi-media virtual memory memory. As all memory mapping for units 610, 620 passes 

controller 630 for the media system. Here, the phrase through multi-media virtual memory controller 630, block 

"multi-media" is used to distinguish the virtual memory 690 can monitor memory accesses and control caching 

controller 630 for the media system from any other virtual accordingly. Block 690 may incorporate a cache memory 

system memory controller (e.g., for generating virtual 65 itself, or may use an external cache. In addition, local 

memory addresses in system memory and a hard disk drive, memory may be used as a "cache" with less frequently 

or a dedicated virtual memory controller for a single device). accessed data relegated to system memory. 



09/28/2004, EAST Version: 1.4.1 



US 6,266,753 Bl 
7 8 

FIG. 8 is a flowchart illustrating the steps in memory The multimedia apparatus of FIG. 7 collects, within one 

allocation by the multimedia virtual mapping system of the structure, different units 751, 752, 753, 754, 755, 756, 757, 

present invention. Block 690 of FIG. 6 may balance memory and 758 along a master transaction bus 710 and data bus 715 

loads dynamically between local memory controller 640 and and slave bus 720 to interface with different types of 

system memory controller 650. In step 801, block 690 5 multimedia data. A PCI slave interface 730 and slave bus 

monitors memory accesses and bus traffic on local and interface 740 may interface slave bus 720 to a host. Note that 

system memory busses. In step 802, if block 690 determines PCI slave interface 730 is technology independent, and thus 

that system memory capacity available for multimedia appli- may incorporate other types of bus interfaces such as the 

cations is running low, block 690 may transfer data via local Advanced Graphics Port (AGP). Thus, PCI slave interface 

memory control 640 in step 803 to local memory and update 10 730, for example, may be substituted with an AGP slave 

the TLB memory map in step 809. interface without departing from the spirit and scope of the 

In step 804, if an inordinate amount of traffic occurs present invention, 

between units 610,620 and system memory (e.g., across and Units 751, 752, 753, 755, 756, 757, and 758 may comprise 

AGP bus or the like) such data may be transferred dynami- a number of specific blocks for manipulating forms of 

cally to local memory via local memory control 640 in step 15 multi-media data. For example, serial interface unit 751 may 

805 to reduce bandwidth requirements on the AGP bus. be provided as a general purpose serial interface which may 

Again, since multi-media virtual memory controller 630 typically be used for audio processing. Serial interface unit 

monitors all memory transactions between units 610,620 751 may pull data from a data buffer in memory and send it 

and memory, it is a simple task to dynamically balance loads to another location (e.g., serial CODEC or via PCI master 

accordingly. The TLB memory map is updated in step 809 20 bus 715 to a USB port). For example, serial interface unit 

and processing returns to monitoring state 801. 751 may take data from a buffer generated by DSP 757 

In step 806, block 690 may determine that data frequently (discussed below) and output that data, through an a/D 

accessed by one or more units 610,620 resides in system converter, to a speaker. Alternately, serial interface unit 751 

memory. If so, such data may be transferred to local memory ma y receive data throu g h a microphone and store it into a 

in step 807. Transferring such data to local memory frees up 25 buffer 10 local or s y stem memory. 

the AGP bus for other duties and allows for faster access of Display timing unit 752 may generate timing signals for 

such data. Again, the TLB allows for faster access of such a video display (e.g., CRT, flat panel display, or the like), 

data. Again, the TLB memory map is updated in step 809 VMI/VIP (Video Media Interface & Video Interface Port) 

such that the entire process appears seamless and transparent interface unit 753 provides a connection between a video 

to units 610,620. 30 device and a graphics controller. Just as serial interface unit 

Dynamic load balancing may also occur if local memory 751 processes audio data, VMI/VIP interface unit 753 may 

becomes full or is near full as illustrated in step 808. Data take data from a frame buffer and output it to a display, or 

may be re-allocated between system resources in a seamless alternately take video input (e.g., from a camera) and store 

and transparent manner in step 810 and the TLB memory 35 it in memory for further processing, 

map updated to reflect these new data locations in step 809. Slave data path 754 may provide a data path between 

There is no need to reprogram each of units 610, 620 to slave bus 720 (and by extension a PCI or AGP bus) and 

achieve such load balancing, and units 610, 620 will not master data bus 715. Slave data path 754 may allow virtual 

even be aware that such data has been moved to new memory manager 760 a path for storing data through PCI 

memory locations. 40 slave bus interface 740 to system memory (not shown). In 

In addition, the use of multi-media virtual memory is addition, slave data path 754 may provide a data path for a 
mapping controller 630 allows the O/S to map the system's host CPU (not shown) to access virtual memory manager 
virtual memory translation table on top of the multi-media 760 through PCI slave interface 740. 
virtual space, allowing units 610,620 and related media Closed-captioning text overlay unit 755 may receive 
software to both work from the same virtual memory 45 closed-captioning data, either within a video or audio data 
mapping plan. Thus, data mapped to system memory stream or as a separate data stream, and generate closed- 
through system memory control 650 may in turn be captioning text data for display. Typically, closed captioning 
re-mapped again by a virtual memory controller for the data may be received from a DVD data stream and stored in 
system, again, in a seamless and transparent manner. system memory. VMI/VIP interface unit may then pull 

FIG. 7 is a block diagram of a multimedia apparatus 50 closed captioning data from system memory and overlay 

which may incorporate the present invention. Whereas FIG. closed captioning text over a video image. 

6 presented a simplified block diagram to aid in the under- MPEG decode unit 756 may decode a stream of MPEG 

standing of the present invention, FIG. 7 presents a detailed data from a number of sources (e.g., DVD or the like) and 

block diagram which also illustrates the preferred embodi- c generate frames of audio and/or video display data as it is 

ment and best mode contemplated of the present invention. 55 needed for display. 

FIG. 7 also better illustrates the number of individual units Digital signal processor (DSP) 757 may comprise one or 

which may be present in a multi-media device. more digital signal processing circuits for processing mul- 

The architecture of the multimedia apparatus of FIG. 7 timedia data. Such DSPs my process, for example, audio or 

may be known as Cirrus Media Architecture, or CMA, and video (MPEG) data or the like, and may comprise, for 

incorporates an Active-X™ accelerator, Active-X™ is a 60 example, a DVD accelerator device. DSP 757 may process 

trademark of Microsoft Corporation and describes a type of one-dimensional (1 -D) video data streams, wavetable MIDI 

graphics rendering more powerful that Microsoft Direct- samples, and the like, and parse audio commands to build 

X™. Active-X™ deals with active movie, MPEG (Motion data streams which may be then output into buffers in main 

Picture Expert Group) DVD (Digital Video, or Digital memory or local memory for later playback (e.g., though 

Versatile Disk) playback, audio streams, video streams, and 65 speakers). 

the like. Such an embodiment may be referred to as a H.26X Video Codec 758 may be provided for encoding 

"streaming engine". and decoding video data coded in the H.261 or H.263 
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formats. H.261 and H.263 are video data standards which For example, a block in system memory is in uncom- 

may be used for video conferencing applications. H.26x pressed form. It may be transferred from system memory 

codec 758 may comprise an encoder/decoder for either one into local memory, and in that step, virtual memory manager 

(or both) of H.261 or H.263 standards, or other similar 760 may compress it. Memory space containing compressed 

standards. 5 data may be referred to as compressed space. The use of 

PLLs 770 may comprise phase locked loop(s) for gener- such compression techniques in and of themselves is well 

ating appropriate multi- media system and bus clocks. Power known in the art. For example, in a graphics controller, it 

management block 780 may be provided to shut down may be difficult to fit a number of surfaces into 4 MB of local 

individual units, portions thereof, or the entire multimedia memory, unless it is compressed. 

system, in order to conserve power. 1Q Talisman™, MPEG, H.261, and H.263, all use a similar 

Each of units 751, 752, 753, 754, 755, 756, 757, and 758 type of discrete cosine transform (DCT) block compression, 

may have the ability to master their own local memory or Quantization factors for each compression type may be 

master to system memory over the PCI (or AGP) bus. Each altered, which may be handled by virtual memory manager 

of units 751, 752, 753, 754 755, 756, 757, and 758 may put 760. 

transaction and requests onto master transaction bus 710. ^ Virtual memory manager 760 may also cache data as 

Virtual memory manager 760 may then determine which discussed above in connection with FIG. 6. Virtual memory 

unit has made a request, and then will partition to the right manager 760 may cache data either internally, or in an 

location in local memory of through PCI (or AGP) slave external cache memory (not shown) such as an SRAM 

interface 730 to main memory (not shown). Virtual memory cache. 

manager 760 is analogous to multi-media virtual memory a number of functions which heretofore had been 
controller 630 of FIG. 6. 20 performed separately in different media units are now per- 
Main memory may comprise a portion of system memory, formed centrally within virtual memory manager 760, a 
or it may be a virtual memory within display memory or number of other advantages may be realized. For example, 
other PCI memory space. System memory is probably the driver ^fiwm for units 751, 752, 5753, 754, 755, 756, 757, 
most likely place for main memory to reside, as the oper- afld ?58 be considcrably simplified, as virtual memory 
ating system (O/S) will try -to keep 'everything m system 25 m mcnt compr ession, and caching need not be 
memory. An operating system such as Windows™ 3.11, addre ^ d bv unit d r iv er software 
Windows™ 95, or Windows™ NT may store data from a * , , • • . ™^ ^ 
data stream in system memory. M discussed above in connection with FIG. 6, another 
However, device drivers may then grab such data and advantage of the present invention is that it allows a number 
move it to a dedicated memory in a particular device (e.g., 30 of metncs (P^'mance measurement indicia) to be placed 
graphics device or the like). In the present invention, by at each ^ entrv - Virlual memorv manager 760 dnver 
virtualizing memory, such memory transactions across buses software may then determine which memory locations are 
may be reduced or eliminated. Rather than have the device being accessed a lot, and dynamically balance memory loads 
driver move data from system memory to local memory, that accordingly without disrupting individual units, 
portion of the system memory may be made a virtual portion 35 It may be appreciated after reading the above disclosure 
of the device memory. that the present invention allows redundant memory map- 
Memory manager 760 may be responsible for keeping ping circuitry for a plurality of units in a multi-media system 
track of virtual memory mapping for all of units 751, I!5 to be combined and centralized. By combining memory 
752, 753, 755, 756, 756, and 758. Memory management 760 mapping functions for a number of units in a multi-media 
may route data through master channel 766 to xDRAM 40 system, the overall size and complexity of the system may 
interface 768 which may interface to local memory (not be reduced. Further, strategic advantages in memory man- 
shown). xDRAM interface 768 may interface to conven- agement are realized by performing virtual memory map- 
tional DRAM, SDRAM, RDRAM, SRAM, or the like. ping for a number of units in one apparatus. Such a system 
Local memory may comprise, for example, display may make heretofore complex and costly multi-media pro- 
memory for a graphics controller, or code space for an audio 45 posals and systems cost effective and practical to implement, 
controller. xDRAM interface 768 may interface to a plural- Although the present invention has been illustrated and 
ity of local memories or memory types, whether on-chip or described in detail, it is clearly understood that the same is 
off chip. by way of illustration and example only and is not to be 
As discussed above in connection with FIG. 6, additional taken by way of limitation, the scope and spirit of the present 
bits may be associated with data addresses for use with the 50 invention being limited only the terms of the appended 
translation table within virtual memory manager 760. Such claims, 
bits may indicate that a memory address does in fact What is claimed is: 

correspond to a physical address. Or, they may reference a 1. A memory manager for use in a multi-media system 

scatter-gather list illustrating how a piece of memory is including a host processor, a system semiconductor memory, 

allocated. Other bits may indicate that data stored at that 55 a plurality of multimedia data processing units, and a local 

address may be compressed (MPEG compression or some semiconductor memory, said memory manager comprising: 

other form of block compression). a memory mapping controller coupled to the plurality of 

Compression may be very relevant especially for archi- multimedia data processing units, the system semicon- 

tectures like Talisman™ where data may be stored in ductor memory, and the local semiconductor memory, 

compressed form, in the present invention, a unit may save 60 the memory mapping controller receiving memory 

data in an uncompressed mode and virtual memory manager access requests from the plurality of multimedia data 

760 may perform all compression and storage functions. In processing units and mapping memory access requests 

addition, virtual memory manager 760 may be accessed by from the plurality of multimedia data processing units 

al l units 7 51, 752, 753, 754, 755, 756, 757, and 758. Texture^ to corresponding portions of the system semiconductor 

-"comprcssionTi image compression, and other types of com- 65 memory and the local semiconductor memory; 

"pression may be managed centrally by virtual memory a memory map storage means, coupled to the memory 

manager 760. mapping controller, for storing a memory map trans- 
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lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and a local semiconductor memory; and 

means for compressing and decompressing data stored in 5 
the system semiconductor memory and the local semi- 
conductor memory, 

wherein said memory map storage means further includes 
means for indicating whether data stored in a portion of 
said system semiconductor memory and the local semi- 30 
conductor memory is in compressed or decompressed 
form, 

wherein said means for compressing and decompressing 
data stored in the system semiconductor memory and 
the local semiconductor memory accesses said memory 35 
map storage means to determine whether data to be 
stored in an accessed portion of said system semicon- 
ductor memory and the local semiconductor memory is 
to be compressed and compresses data written to the 
portion of said system semiconductor memory and the 20 
local semiconductor memory, and 

wherein said means for compressing and decompressing 
data stored in the system semiconductor memory and 
the local semiconductor memory accesses said memory 25 
map storage means to determine whether data to be 
retrieved from an accessed portion of said system 
semiconductor memory and the local semiconductor 
memory is compressed and decompresses data read 
from the portion of said system semiconductor memory 3Q 
and the local semiconductor memory, 

2. The memory manager of claim 1, wherein said memory 
mapping controller maps memory access requests from the 
plurality of multimedia data processing units to portions of 
the system semiconductor memory and the local semicon- 35 
ductor memory according to the memory map stored in said 
memory map storage means. 

3. The memory manager of claim 2, wherein said plurality 
of multimedia data processing units comprise video, audio, 
and data processing units for processing types of multimedia 4Q 
data. 

4. The memory manager of claim 1, further comprising: 
a data cache, coupled to the memory controller, for 

caching data frequently accessed by one or more of the 
plurality of multimedia data processing units. 45 

5. A memory manager for use in a multi-media system 
including a host processor, a system semiconductor memory, 
a plurality of multimedia data processing units, and a local 
semiconductor memory, said memory manager comprising: 

a memory mapping controller coupled to the plurality of 50 
multimedia data processing units, the system semicon- 
ductor memory, and the local semiconductor memory, 
the memory mapping controller receiving memory 
access requests from the plurality of multimedia data 
processing units and mapping memory access requests 55 
from the plurality of multimedia data processing units 
to corresponding portions of the system semiconductor 
memory and the local semiconductor memory; 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory map trans- 60 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and a local semiconductor memory, 

wherein said memory mapping controller mass memory 65 
access requests from the plurality of multimedia data 
processing units to portions of the system semiconduc- 
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tor memory and the local semiconductor memory 
according to the memory map stored in said memory 
map storage means, and 
wherein said memory controller monitors frequency of 
access of data in said portions of system semiconductor 
memory and said portions of local semiconductor 
memory and dynamically transfers frequently accessed 
data from system semiconductor memory to local semi- 
conductor memory and updates the memory map to 
indicate new locations for transferred data. 

6. A memory manager for use in a multi-media system 
including a host processor, a system semiconductor memory, 
a plurality of multimedia data processing units, and a local 
semiconductor memory, said memory manager comprising: 

a memory mapping controller coupled to the plurality of 
multimedia data processing units, the system semicon- 
ductor memory, and the local semiconductor memory, 
the memory mapping controller receiving memory 
access requests from the plurality of multimedia data 
processing units and mapping memory access requests 
from the plurality of multimedia data processing units 
to corresponding portions of the system semiconductor 
memory and the local semiconductor memory; 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory map trans- 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and a local semiconductor memory, 

wherein said memory mapping controller mass memory 
access requests from the plurality of multimedia data 
processing units to portions of the system semiconduc- 
tor memory and the local semiconductor memory 
according to the memory map stored in said memory 
map storage means, and 

wherein said memory controller monitors data transfer 
bandwidth of the system semiconductor memory and 
the local semiconductor memory and dynamically 
transfers frequently accessed data between system 
semiconductor memory and local semiconductor 
memory to reduce data transfer bandwidth of the sys- 
tem semiconductor memory and the local semiconduc- 
tor memory and updates the memory map to indicate 
new locations for transferred data. 

7. A memory manager for use in a multi-media system 
including a host processor, a system semiconductor memory, 
a plurality of multimedia data processing units, and a local 
semiconductor memory, said memory manager comprising: 

a memory mapping controller coupled to the plurality of 
multimedia data processing units, the system semicon- 
ductor memory, and the local semiconductor memory, 
the memory mapping controller receiving memory 
access requests from the plurality of multimedia data 
processing units and mapping memory access requests 
from the plurality of multimedia data processing units 
to corresponding portions of the system semiconductor 
memory and the local semiconductor memory; 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory map trans- 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and a local semiconductor memory, 

wherein said memory mapping controller mass memory 
access requests from the plurality of multimedia data 
processing units to portions of the system semiconduc- 
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tor memory and the local semiconductor memory 
according to the memory map stored in said memory 
map storage means, and 
wherein said memory controller monitors available 
memory capacity of the system semiconductor memory 
and the local semiconductor memory and dynamically 
transfers data between system semiconductor memory 
and local semiconductor memory to increase available 
memory capacity of one of the system semiconductor 
memory and the local semiconductor memory and 
updates the memory map to indicate new locations for 
transferred data. 

8. A multi-media apparatus for processing multi-media 
data, comprising: 

a plurality of multimedia data processing units, each for 
processing multi-media data and reading and storing 
data in memory; 

a memory mapping controller coupled to the plurality of 
multimedia data processing units and to a system 
semiconductor memory and a local semiconductor 
memory, the memory mapping controller receiving 
memory access requests from the plurality of multime- 
dia data processing units and mapping memory access 
requests from the plurality of multimedia data process- 
ing units to corresponding portions of the system 
semiconductor memory and the local semiconductor 
memory; and 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory map trans- 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and the local semiconductor memory, 

wherein said memory manager further comprises: 

means for compressing and decompressing data stored in 
the system semiconductor memory and the local semi- 
conductor memory, 

wherein said memory map storage means further includes 
means for indicating whether data stored in a portion of 
the system semiconductor memory and the local semi- 
conductor memory is in compressed or decompressed 
form, 

wherein said means for compressing and decompressing 
data stored in the system semiconductor memory and 
the local semiconductor memory accesses said memory 
map storage means to determine whether data to be 
stored in an accessed portion of the system semicon- 
ductor memory and the local semiconductor memory is 
to be compressed and compresses data written to the 
portion of the system semiconductor memory and the 
local semiconductor memory, and 

wherein said means for compressing and decompressing 
data stored in the system semiconductor memory and 
the local semiconductor memory accesses said memory 
map storage means to determine whether data to be 
retrieved from an accessed portion of the system semi- 
conductor memory and the local semiconductor 
memory is compressed and decompresses data read 
from the portion of the system semiconductor memory 
and the local semiconductor memory. 

9, The multi-media apparatus of claim 8, wherein said 
memory mapping controller maps memory access requests 
from the plurality of multimedia data processing units to 
portions of the system semiconductor memory and the local 
semiconductor memory according to the memory map 
stored in said memory map storage means. 
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10. The multi-media apparatus of claim 9, wherein said 
plurality of multimedia data processing units comprise 
video, audio, and data processing units for processing types 
of multimedia data, 

11. The multi-media apparatus of claim 8, wherein said 
memory manager further comprises: 

a data cache, coupled to the memory controller, for 
caching data frequently accessed by one or more of the 
plurality of multimedia data processing units. 

12. A multi-media apparatus for processing multi-media 
data, comprising: 

a plurality of multimedia data processing units, each for 
processing multi-media data and reading and storing 
data in memory; > 

a memory mapping controller coupled to the plurality of 
multimedia data processing units and to a system 
semiconductor memory and a local semiconductor 
memory, the memory mapping controller receiving 
memory access requests from the plurality of multime- 
dia data processing units and mapping memory access 
requests from the plurality of multimedia data process- 
ing units to corresponding portions of the system 
semiconductor memory and the local semiconductor 
memory; and 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory may trans- 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and the local semiconductor memory, 

wherein said memory mapping controller maps memory 
access requests from the plurality of multimedia data 
processing units to portions of the system semiconduc- 
tor memory and the local semiconductor memory 
according to the memory map stored in said memory 
map storage means, and 

wherein said memory controller monitors frequency of 
access of data in said portions of system semiconductor 
memory and said portions of local semiconductor 
memory and dynamically transfers frequently accessed 
data from system semiconductor memory to local semi- 
conductor memory and updates the memory map to 
indicate new locations for transferred data, 

13. A multi-media apparatus for processing multi-media 
data, comprising: 

a plurality of multimedia data processing units, each for 
processing multi-media data and reading and storing 
data in memory; 

a memory mapping controller coupled to the plurality of 
multimedia data processing units and to a system 
semiconductor memory and a local semiconductor 
memory, the memory mapping controller receiving 
memory access requests from the plurality of multime- 
dia data processing units and manning memory access 
requests from the plurality of multimedia data process- 
ing units to corresponding portions of the system 
semiconductor memory and the local semiconductor 
memory; and 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory may trans- 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and the local semiconductor memory, 

wherein said memory mapping controller maps memory 
access requests from the plurality of multimedia data 
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processing units to portions of the system semiconduc- 
tor memory and the local semiconductor memory 
according to the memory map stored in said memory 
map storage means, and 
wherein said memory controller monitors data transfer 
bandwidth of the system semiconductor memory and 
the local semiconductor memory and dynamically 
transfers frequently accessed data between system 
semiconductor memory and local semiconductor 
memory to reduce data transfer bandwidth of the sys- 
tem semiconductor memory and the local semiconduc- 
tor memory and updates the memory map to indicate 
new locations for transferred data. 

14. A multi-media apparatus for processing multi-media 
data, comprising: 

a plurality of multimedia data processing units, each for 
processing multi-media data and reading and storing 
data in memory; 

a memory mapping controller coupled to the plurality of 
multimedia data processing units and to a system 
semiconductor memory and a local semiconductor 
memory, the memory mapping controller receiving 
memory access requests from the plurality of multime- 
dia data processing units and mapping memory access 
requests from the plurality of multimedia data process- 
ing units to corresponding portions of the system 
semiconductor memory and the local semiconductor 
memory; and 

a memory map storage means, coupled to the memory 
mapping controller, for storing a memory map trans- 
lating memory addresses for memory space for the 
plurality of multimedia data processing units to physi- 
cal memory addresses in the system semiconductor 
memory and the local semiconductor memory, 

wherein said memory mapping controller maps memory 
access requests from the plurality of multimedia data 
processing units to portions of the system semiconduc- 
tor memory and the local semiconductor memory 
according to the memory map stored in said memory 
map storage means, and 

wherein said memory controller monitors available 
memory capacity of the system semiconductor memory 
and the local semiconductor memory and dynamically 
transfers data between system semiconductor memory 
and local semiconductor memory to increase available 
memory capacity of one of the system semiconductor 
memory and the local semiconductor memory and 
updates the memory map to indicate new locations for 
transferred data. 

15. A method for managing memory for a plurality of data 
processing multimedia data processing units coupled to the 
system semiconductor memory and the local semiconductor 
memory, said method comprising the steps of: 

receiving, in a memory mapping controller coupled to the 
plurality of multimedia data processing units and to the 
system semiconductor memory and the local semicon- 
ductor memory, memory access requests from the plu- 
rality of multimedia data processing units, 

translating memory addresses for memory space for the 
plurality of multimedia data processing units to corre- 
sponding physical memory addresses in the system 
semiconductor memory and the local semiconductor 
memory according to a memory map, and 

mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory, 
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reading, from the memory map, data indicating whether 
data stored in a portion of the system semiconductor 
memory and the local semiconductor memory is in 
compressed or decompressed form, 

determining, from the memory map whether data to be 
stored in an accessed portion of the system semicon- 
ductor memory and the local semiconductor memory is 
to be compressed, and 

compressing data written to the portion of the system 
semiconductor memory and the local semiconductor 
memory if the data is determined to be compresses. 

16. The method of claim 15, wherein the step of mapping 
memory access requests comprises the step of: 

mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory according to the memory map stored in 
the memory map storage means. 

17. The method of claim 16, wherein the plurality of 
multimedia data processing units comprise video, audio, and 
data processing units for processing types of multimedia 
data. 

18. The method of claim 15, further comprising the step 

of: 

caching data frequently accessed by one or more of the 
plurality of multimedia data processing units in a data 
cache coupled to the memory controller. 

19. A method for managing memory for a plurality of data 
processing multimedia data processing units coupled to the 
system semiconductor memory and the local semiconductor 
memory, said method comprising the steps of: 

receiving, in a memory mapping controller coupled to the 
plurality of multimedia data processing units and to the 
system semiconductor memory and the local semicon- 
ductor memory, memory access requests from the plu- 
rality of multimedia data processing units, 

translating memory addresses for memory space for the 
plurality of multimedia data processing units to corre- 
sponding physical memory addresses in the system 
semiconductor memory and the local semiconductor 
memory according to a memory map, 

mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory, 

monitoring, in the memory controller, frequency of access 
of data in the portions of system semiconductor 
memory and the portions of local semiconductor 
memory, 

dynamically transferring frequently accessed data from 
system semiconductor memory to local semiconductor 
memory, and 

updating the memory map to indicate new locations for 
transferred data, 

wherein the step of mapping memory access requests 
comprises the step of: 

mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory according to the memory may stored in 
the memory map storage means. 

20. A method for managing memory for a plurality of data 
processing multimedia data processing units coupled to the 
system semiconductor memory and the local semiconductor 
memory, said method comprising the steps of: 
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receiving, in a memory mapping controller coupled to the 
plurality of multimedia data processing units and to the 
system semiconductor memory and the local semicon- 
ductor memory, memory access requests from the plu- 
rality of multimedia data processing units, 5 

translating memory addresses for memory space for the 
plurality of multimedia data processing units to corre- 
sponding physical memory addresses in the system 
semiconductor memory and the local semiconductor 
memory according to a memory map, 30 

mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory, 

monitoring, in the memory controller, data transfer band- 
width of the system semiconductor memory and the 
local semiconductor memory, 

dynamically transferring frequently accessed data 
between system semiconductor memory and local 2 o 
semiconductor memory to reduce data transfer band- 
width of the system semiconductor memory and the 
local semiconductor memory, and 

updating the memory map to indicate new locations for 
transferred data, 25 

wherein the step of mapping memory access requests 
comprises the step of: 

mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 30 
ductor memory according to the memory map stored in 
the memory map storage means. 

21. A method for managing memory for a plurality of data 
processing multimedia data processing units coupled to the 
system semiconductor memory and the local semiconductor 35 
memory, said method comprising the steps of: 
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receiving, in a memory mapping controller coupled to the 
plurality of multimedia data processing units and to the 
system semiconductor memory and the local semicon- 
ductor memory, memory access requests from the plu- 
rality of multimedia data processing units, 
translating memory addresses for memory space for the 
plurality of multimedia data processing units to corre- 
sponding physical memory addresses in the system 
semiconductor memory and the local semiconductor 
memory according to a memory map, 
mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory, 
monitoring, in the memory controller, available memory 
capacity of the system semiconductor memory and the 
local semiconductor memory, 
dynamically transferring data between system semicon- 
ductor memory and local semiconductor memory to 
increase available memory capacity of one of the 
system semiconductor memory and the local semicon- 
ductor memory, and 
updating the memory map to indicate new locations for 

transferred data, 
wherein the step of mapping memory access requests 

comprises the step of: 
mapping memory access requests from the plurality of 
multimedia data processing units to portions of the 
system semiconductor memory and the local semicon- 
ductor memory according to the memory may stored in 
the memory map storage means. 

***** 
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